A figure of merit for the analysis of spoken dialog systems

In this paper, a single metric, which we will call the figure of merit , for the quantitative analysis and comparison of spoken dialog systems is introduced. This figure of merit is the product of the weighted dialog accuracy (expressed as the rate of success) and the weighted dialog efficiency (expressed as the average number of concepts per turn). Actually, it is highly desirable to have a quick and accurate dialog. However, these two requirements are conflicting. That is, an improvement in efficiency is accomplished at the expense of accuracy or vice versa. This makes difficult to compare two different spoken dialog systems or tune a particular system. We believe that this figure of merit would avoid those difficulties. To illustrate its use, we consider spoken dialog systems with different dialog strategies and compare them by performing quantitative analysis based on the finate state models of information items using the proposed metric.